Bayesian Modelling for Data Analysis and Learning from Data

These notes provide clarifying remarks and definitions complementing the course Bayesian Modelling for Data Analysis and Learning from Data, to be held at IK 2006. 1 Bayesian Statistics for Machine Learning In this Section we define and motivate basic terms of probability and Bayesian statistics relevant for Machine Learning. The linear model is introduced, the notion of complexity control via Occam’s razor is motivated. 1.1 Machine Learning: Managing and Quantifying Uncertainty Machine Learning (ML) is a hybrid of Statistics and algorithmic Computer Science (CS). In this course, we will be mostly concerned with the statistical side. Statistics is about managing and quantifying uncertainty. Uncertainty may arise due to many different reasons, for example: • Measurement noise: Measurements of physical processes are always subject to inaccuracies. Sometimes, low quality data may be obtained more economically. Data items may be missing • Model uncertainty: Models are almost never exact, we abstract away complexity in order to allow for predictions to be feasible • Parameter uncertainty: Variables in a model can never be identified exactly and without doubt from finite data The calculus of uncertainty is probability theory, see [2] for a good introduction. Some phenomenon of interest is mapped to a model, being a set of random variables and probabilistic relationships between them. Variables are observed or latent (unobserved). To give an example, consider the linear model